feat(cli): --dry-run, --base-url, --timeout, and the load report in the artifact - #94
Merged
Merged
Conversation
…he artifact A dry run loads and checks the dataset, builds the adapter, and applies the cost guard through the same plan the run starts from, then prints the case count, endpoint, timeout, semantics, and estimate, and sends nothing. --base-url and --timeout reach an adapter that takes them, and the timeout is recorded beside the endpoint. --semantics now overrides any adapter that takes it, and the report says the operator set it. An option an adapter does not take is refused by name, read from its signature. An artifact stores a LoadSummary, so plumbline report rebuilds the Dataset section it used to drop. Fixes #58. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #58.
--dry-run.execute.plan()prices the run and applies the guard without sending anything, andexecute.run()now starts from it too. So the dry run refuses exactly what the run would refuse (a case limit, an unpriced model under--max-cost-usd, an estimate over the limit), with the same messages. When the run would go ahead, it prints the plan to stdout and exits 0 without writing a file:With a priced model, the cost line names the estimate, the pricing entry and the date it was read.
--base-urland--timeout. These are passed to adapters that accept them:typesafe_wireandgenerative. The endpoint was already recorded in the artifact and part of the cache key. The timeout is now recorded too, astimeout_seconds.--timeoutmust be above 0.--semantics. It now overrides the declared semantics for any adapter that accepts it, not only the mock. The artifact recordssemantics_set_by: operator, and the report adds a line saying the operator set it.Refused options. An option an adapter doesn't accept is now refused by name before the adapter is built, using the new
registry.accepts(), which reads the adapter's signature. For example,the generative adapter does not take --semantics. Before, the message listed every option that had been passed.The load report in the artifact. The new
LoadSummaryholds the load's counts, refusals, notes, and the rows excluded by question type, without the cases.RunResult.loadstores it.LoadReport.statement()now goes throughLoadSummary.statement(), so both print the same sentence.plumbline reportpasses the stored load when every artifact covers one dataset and they all stored the same load. Otherwise it leaves the section out rather than print one run's counts over another's figures. Artifacts written before this change still load, just without the section.Docs. README, CONTRIBUTING and
.env.examplesaid a new endpoint was "a config entry", which had no mechanism behind it. They now say--base-url. README also suggests--dry-runbefore a paid run.Tests. The new
tests/test_cli_run_options.pycovers:typesafe_wiredry run, and the dummy key never appears in the output--timeout 0is refuseddocs/example-report.mdis unchanged, since its command passes none of the new options. The full gate, the site build and every site check, and the prose checks all pass.🤖 Generated with Claude Code